The literature on Inverse Reinforcement Learning (IRL) typically assumes thathumans take actions in order to minimize the expected value of a cost function,i.e., that humans are risk neutral. Yet, in practice, humans are often far frombeing risk neutral. To fill this gap, the objective of this paper is to devisea framework for risk-sensitive IRL in order to explicitly account for a human'srisk sensitivity. To this end, we propose a flexible class of models based oncoherent risk measures, which allow us to capture an entire spectrum of riskpreferences from risk-neutral to worst-case. We propose efficientnon-parametric algorithms based on linear programming and semi-parametricalgorithms based on maximum likelihood for inferring a human's underlying riskmeasure and cost function for a rich class of static and dynamicdecision-making settings. The resulting approach is demonstrated on a simulateddriving game with ten human participants. Our method is able to infer and mimica wide range of qualitatively different driving styles from highly risk-averseto risk-neutral in a data-efficient manner. Moreover, comparisons of theRisk-Sensitive (RS) IRL approach with a risk-neutral model show that the RS-IRLframework more accurately captures observed participant behavior bothqualitatively and quantitatively, especially in scenarios where catastrophicoutcomes such as collisions can occur.
展开▼
